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ABSTRACT 

This  thesis  quantitatively  analyzes  the  factors  that  affect  weapon 
system  cost  growth  after  Milestone  II.  The  data  from  nine  weapon  systems 
was  reconstructed  by  the  Army  and  Navy  from  Selected  Acquisition  Reports 
(SARs)  with  the  cost  variances  reclassified  into  a  new  categorization 
system  to  more  readily  determine  the  causes  of  cost  growth.  Each  cost 
variance  was  classified  as  to  whether  it  was  attributable  to  a  mistake  in 
the  cost  estimating  process  or  a  post-Milestone  II  decision,  with  further 
classification  into  subcategories  for  a  more  detailed  analysis.  The  cost 
variances  were  divided  by  the  Milestone  II  Decision  Estimate  (DE)  to  form 
a  cost  growth  ratio  (CGR) .  The  findings  reveal  that  the  Department  of 
Defense  has  about  10.8%  cost  growth  in  the  procurement  process.  Cost 
growth  due  to  decisions  outweigh  mistakes  by  a  factor  of  2.3:1.  A 
majority  of  the  mistake  cost  growth  is  due  to  errors  in  the  estimation  of 
production  costs.  A  majority  of  the  decision  cost  growth  is  due  to 
schedule  slippage.  Low  cost  systems  have  2.4  times  as  much  mistake  cost 
growth  as  high  cost  systems.  Newer  missile  systems  have  significantly 
less  mistake  cost  growth  when  compared  to  other  systems.  Lastly,  the  Army 
and  Navy  have  approximately  equal  cost  growth  on  their  newer  systems . 
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EXECUTIVE  SUMMARY 

A  common  perception  among  the  public,  in  Congress,  and  even  within  the 
defense  establishment  is  that  weapon  system  cost  growth  results  from  poor 
managerial  practices  combined  with  an  inability  to  accurately  estimate 
weapons  system  costs.  This  oversimplification  of  a  very  complex  problem 
is  at  the  forefront  of  discussion  now  that  reduced  budgets,  military 
downsizing,  and  decreased  procurement  are  realities. 

Many  cost  growth  studies  occurred  in  the  early  1980s  in  response  to 
the  military  buildup  which  began  in  1980.  Very  little  research  has  been 
done  since  then  on  the  new  high  technology  systems.  This  lack  of  adequate 
research  combined  with  tighter  military  budgets  shows  a  need  for  new 
research  on  the  factors  affecting  cost  growth.  The  objective  of  this 
research  is  not  to  predict  or  model  particular  systems,  but  to  determine 
those  factors  that  affect  the  cost  growth  of  the  Department  of  Defense 
(DoD)  as  a  whole. 

In  order  to  more  readily  identify  areas  of  cost  growth,  it  is 
necessary  to  classify  the  cost  variances  into  a  more  useable  and  more 
detailed  categorization  system.  The  new  categorization  system 
differentiates  between  decisions  made  that  would  knowingly  increase  weapon 
system  cost  and  mistakes  that  misestimated  aspects  in  the  process  which, 
if  estimated  correctly,  would  not  have  resulted  in  cost  growth.  The  new 
categories  also  partition  these  two  broad  categories  into  six 
subcategories  each  to  give  insight  into  the  most  likely  areas  for  cost 
growth  to  occur. 

The  data  was  analyzed  to  determine  if  factors  such  as  system  cost, 
service,  and  type  of  system  affect  cost  growth.  This  was  done  to  identify 
specific  areas  in  the  cost  estimation  and  procurement  processes  that  need 
attention  and  to  determine  the  probable  causes  of  these  differences. 


The  findings  reveal  that  the  Department  of  Defense  has  experienced 
approximately  10.8%  cost  growth  in  the  procurement  process  after  Milestone 
II,  which  is  lower  than  might  be  expected.  Studies  done  during  the  early 
1980s  report  that  the  average  total  cost  growth  of  weapon  systems  was 
between  20%  and  30%.  However,  the  time  of  those  studies  is  relevant  to 
the  difference  in  findings.  Three  major  changes  have  occurred  in  the  last 
decade  to  lower  weapon  system  cost  growth.  First,  estimators  have 
developed  a  large  computerized  cost  data  base  to  more  accurately  predict 
costs.  Second,  high  technology  systems,  that  were  brand  new  in  1980,  have 
been  developed  and  produced  so  that  they  are  now  much  easier  to  estimate. 
Lastly,  the  early  studies  resulted  in  a  determined  effort  by  the 
Department  of  Defense  to  control  cost  growth. 

Mistakes  made  in  the  estimation  of  system  costs  make  up  30.6%  of  the 
total  cost  growth  of  a  system  while  decisions  make  up  69.4%  of  the  total 
cost  growth.   Therefore  decisions  outweigh  mistakes  by  a  margin  of  2.3:1. 

A  majority  of  the  mistake  cost  growth  is  attributable  to  mistakes  in 
estimating  the  cost  of  production.  However,  mistake  cost  growth  averages 
only  3.3%  and  is  not  nearly  as  significant  as  controlling  the  cost  growth 
due  to  decisions. 

A  majority  of  the  decision  cost  growth  is  attributable  to  schedule 
slippage.  Some  of  this  schedule  slippage  can  be  attributed  to  decisions 
to  change  the  design  and  performance  requirements  of  the  system,  while  the 
remaining  amount  is  unexplained.  Further  study  concentrating  on  the 
causes  of  schedule  slippage  needs  to  be  done. 

The  low  cost  systems  have  a  higher  mistake  cost  growth  when  compared 
to  high  cost  systems  by  a  margin  of  2.4:1.  Two  possible  explanations 
exist  for  this  effect.  First,  estimators  do  not  take  as  much  care  in 
estimating  the  costs  of  low  cost  systems  since  they  are  not  as  visible  on 
an  oversight  level.  The  second  reason  concerns  the  nature  of  high  cost 
and  low  cost  systems.  A  majority  of  high  cost  systems  are  large  platforms 
which  have  a  majority  of  their  production  cost  resulting  from  large 
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components  or  similar  components  that  have  been  used  before.  This  makes 
the  estimation  of  the  production  costs  of  these  large  components  fairly- 
accurate.  Therefore,  the  mistakes  in  estimating  the  smaller  and 
relatively  cheaper  items  would  be  masked  in  high  cost  systems  by  these 
high  cost  items.  Therefore,  the  difference  between  the  low  cost  and  high 
cost  system  mistake  cost  growth  is  probably  not  as  significant  as  it 
appears  to  be . 

The  significant  difference  between  the  cost  growth  of  new  missile 
systems  and  other  weapon  systems  is  that  the  mistake  cost  growth  for  new 
missile  systems  is  significantly  less  than  that  of  other  systems, 
specifically  in  estimating  the  cost  of  production.  This  is  due  to  the 
evolutionary  nature  of  missile  systems.  The  guidance,  propulsion,  and 
warhead  systems  of  newer  missiles  is  generally  a  modernization  or 
modification  of  a  previous  system.  This  would  tend  to  increase  the 
accuracy  of  missile  system  estimates  over  other  weapon  systems.  Previous 
studies  show  that  missile  systems  have  a  much  higher  cost  growth  than  do 
other  systems.  These  studies  included  much  older  missile  systems.  The 
evolutionary  process  discussed  above  had  not  yet  developed  and  therefore 
a  higher  cost  growth  could  have  resulted. 

The  Army  and  Navy  have  effectively  equal  cost  growths.  A  majority  of 
early  1980s  studies  reported  that  the  Army  had  a  significantly  higher  cost 
growth  than  the  Navy.  This  was  attributed  to  the  modernization  of  the 
Army.  This  study  concentrated  on  more  recent  weapons  systems  and  did  not 
find  this  to  be  the  case.  The  most  obvious  explanation  for  this  effect  is 
that  the  Army  has  completed  its  modernization  and  is  now  proficient  at 
controlling  the  costs  of  its  weapon  systems. 

These  results  are  promising  in  that  it  appears  that  over  the  last 
decade,  the  DoD  has  gained  control  over  cost  growth.  It  is  still 
necessary  to  determine  and  correct  the  cause  of  schedule  slippage,  but,  on 
the  whole,  the  cost  growth  problem  is  being  solved. 
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I .   INTRODUCTION 

A  common  perception  among  the  public,  in  Congress,  and  even  within 
the  defense  establishment  is  that  weapons  system  cost  growth  results 
from  poor  managerial  practices  combined  with  an  inability  to  accurately 
estimate  weapons  system  costs.   This  oversimplification  of  a  very 
complex  problem  is  at  the  forefront  of  discussion  now  that  reduced 
budgets,  military  downsizing,  and  decreased  procurement  are  realities. 

Many  studies  have  shown  that  most  of  a  weapons  system' s  procurement 
cost  growth  occurs  after  it  enters  Engineering  and  Manufacturing 
Development  (EMD)  but  prior  to  Full  Scale  Production  [Ref .  1] .   This 
cost  growth  will  be  magnified  if  the  United  States  attempts  to  maintain 
its  technological  edge  in  the  reduced  budget  environment  by  developing 
technology  and  then  "shelving  it"  until  needed.   This  will  result  in 
increased  scrutiny  by  the  Congress  on  cost  growth  and  its  control . 

Most  of  the  research  on  cost  growth  occurred  prior  to  the  1980' s 
military  buildup.   Some  research  focused  on  single  system  cost  growth 
(e.g.,  just  the  F-14  [Ref.  2]).   This  was  done  to  identify  and  change 
managerial  and  estimating  problems  within  a  particular  weapons  system 
already  identified  as  having  high  cost  growth.   The  remaining  research 
used  the  Selected  Acquisition  Report  (SAR)  categories  for  a  basis  [Ref 
3] .   In  the  SAR,  each  cost  variance  is  categorized  as  an  estimating, 
economic,  quantity,  schedule,  support  or  other  change  from  the  Milestone 
II  baseline  estimate.   Unfortunately,  these  categories  were  not 
uniformly  used  across  different  systems  to  classify  cost  variances, 
making  any  cross-system  analysis  using  these  categories  of  limited 
validity.   Furthermore,  the  criteria  for  classifying  a  cost  variance 
into  one  of  these  SAR  categories  changed  in  the  mid-1980' s  and  as  a 
result,  comparisons  across  this  time  are  not  valid.   Therefore,  none  of 


the  relatively  new  high  technology  systems  can  be  analyzed  using  a  time 
series  approach. 

As  a  result  of  this  lack  of  adequate  data  on  current  technology 
combined  with  tighter  military  budgets,  new  research  on  the  factors 
affecting  cost  growth  is  required.   It  will  be  necessary  to  form  new 
categorization  criteria  for  cost  variances  in  order  to  break  the  cost 
variances  into  a  more  useful  structure.   The  objective  of  this  research 
is  not  to  predict  or  model  particular  systems,  but  to  determine  those 
factors  that  affect  the  cost  growth  of  the  Department  of  Defense  as  a 
whole.   The  quantitative  analysis  of  this  data  is  designed  to  yield 
descriptive  information  on  those  areas  in  the  procurement  process  where 
cost  growth  is  occurring.   This  information  can  be  used  by  Program 
Managers  and  cost  estimators  to  evaluate  their  cost  methodology  and 
refine  their  cost  estimating  techniques.   This  should  ultimately  yield 
lower  procurement  cost  growth. 


II.   BACKGROUND 

A.   REVIEW  OF  THE  ACQUISITION  PROCESS 

The  purpose  of  this  chapter  is  to  provide  a  brief  review  of  the 
weapons  system  acquisition  process  prior  to  a  discussion  of  research 
methodology  used  and  why  data  from  specific  points  in  the  acquisition 
process  was  used. 

The  acquisition  process  is  broken  down  into  five  phases  with  the 
transition  between  phases  being  marked  by  a  milestone.   Decision 
authorities  determine  at  each  milestone  whether  the  results  of  the 
previous  phase  warrant  continuation  into  the  next  acquisition  phase. 
Department  of  Defense  Instruction  5000.2  (current  version  dated  February 
1991)  describes  the  process  and  discusses  the  objectives  of  each  phase 
and  milestone  in  the  acquisition  process.   Systems  within  the 
acquisition  process  do  not  necessarily  follow  this  exact  process. 
However,  each  system  must  ultimately  gain  approval  at  each  milestone  in 

this  process. 

1.  Mission  Need 

Initially,  a  determination  of  mission  need  is  decided  upon,  by 
various  sources,  based  upon  continuing  assessments  of  the  military 

requirements  and  capabilities. 

2.  Milestone  0,  Concept  Studies  Approval 

Decision  authorities  determine  if  a  study  of  alternatives  is 
warranted  and  determine  the  alternatives  to  be  evaluated.   If  approval 

is  granted,  the  system  proceeds  into  Phase  0. 

3.  Phase  0,  Concept  Exploration  and  Definition 

Various  alternatives  are  evaluated.   The  most  promising  systems 
satisfying  the  mission  requirements  are  evaluated  to  determine  high  risk 
areas  and  initial  objectives  for  program  cost,  schedule,  and 
performance . 


4.  Milestone  I,  Concept  Demonstration  Approval 

Decision  authorities  determine  if  a  new  acquisition  program  is 
needed  based  on  the  results  of  Phase  0.   A  baseline  for  program  cost, 
schedule,  and  performance  is  established.   Approval  at  Milestone  I 

authorizes  the  program  to  move  into  Phase  I. 

5.  Phase  I,  Demonstration  and  Validation 

Critical  design  and  performance  characteristics  are  thoroughly- 
evaluated.   Schedule,  cost,  and  performance  objectives  are  refined. 

6.  Milestone  II,  Development  Approval 

If  the  results  of  Phase  I  warrant  continuation,  a  Developmental 
Baseline  for  cost,  schedule  and  performance  for  the  program  is  approved. 
This  includes  the  Developmental  Estimate  (DE)  which  defines  the 
objective  total  system  cost.   This  estimate  is  the  refined  total  cost 
from  Milestone  I.   Milestone  II  approval  authorizes  program  continuation 

into  Phase  II. 

7.  Phase  II,  Engineering  and  Manufacturing  Development 

The  system  enters  full  scale  engineering  development.   A  cost 
effective,  producible  design  is  developed  to  validate  the  production 
process.   Tests  are  conducted  to  verify  that  system  performance 

satisfies  performance  requirements. 

8.  Milestone  III,  Production  Approval 

A  production  baseline  for  cost,  schedule  and  performance  using 
the  revised  data  from  Phase  II  is  established.   The  system  now  proceeds 

into  Phase  III. 

9.  Phase  III,  Production  and  Deployment/Phase  IV,  Operations  and 

Support 

An  efficient  production  capability  along  with  its  associated 
support  base  is  established  to  achieve  operational  capability  satisfying 
the  mission  need.   Once  initial  systems  are  fielded,  system  monitoring, 
support  and  problem  correction  continue  throughout  system  lifetime. 


10.      Milestone  IV,  Major  Modification  Approval 

Decision  authorities  determine  if  a  major  modification  or 
change  to  the  system  is  warranted  and  if  so,  how  to  implement  this 
change  in  the  most  effective  manner. 

B.   SELECTED  ACQUISITION  REPORT  (SAR) 

The  data  used  is  derived  from  the  Selected  Acquisition  Report  (SAR) . 
Comprehensive  instructions  for  the  SAR  are  contained  in  Department  of 
Defense  Manual   5000. 2M.   The  SAR  is  a  comprehensive  report  designed  to 
provide  Defense  Department  officials  with  cost  and  management 
information  on  each  major  weapons  system.   The  SARs  are  submitted  to  the 
Congress  to  allow  the  Armed  Services  Committees  to  monitor  the 
Department's  progress  in  meeting  its  procurement  plans,  and  to  provide 
an  early  warning  of  emerging  cost  problems . 

The  SARs  are  compilations  of  status  reports  from  the  Program 
Managers  responsible  for  major  defense  acquisition  programs.   They 
provide  each  Program  Manager's  latest  estimates  of  progress  in  achieving 
key  goals  with  respect  to  performance,  schedule  and  cost.   The  most 
recent  estimates  are  recorded  in  then  year  dollars  and  in  constant  base 
year  dollars,  with  the  base  year  generally  being  the  Milestone  II  year. 

SARs  are  required  by  Department  of  Defense  Manual  5000. 2M  to  be 
submitted  for  programs  that  have  been  designated  by  the  Secretary  of 
Defense  as  major  systems  or  are  estimated  to  cost  more  than  $200  million 
for  research,  development,  test,  and  evaluation  or  more  than  $1  billion 
for  procurement.   Highly  classified  programs  are  excluded  from  this 
requirement.   SARs  are  prepared  on  an  exception  basis  for  the  first, 
second,  and  third  quarter  of  each  year,  with  a  comprehensive  report  for 
the  fourth  quarter  ending  December  31.   The  cost  data  in  the  December 
SARs  are  expected  to  correspond  to  data  in  the  President's  annual  budget 
submitted  to  the  Congress  in  January. 


1.   SAR  Categories 

The  SAR  reports  the  Program  Manager's  most  recent  best  cost 
estimate.   The  initial  estimate  listed  in  the  SAR  is  the  baseline 
estimate  or  developmental  estimate  (DE) .   The  DE  is  generally  made  at 
Milestone  II  in  the  acquisition  process.   The  SAR  records  all  changes  in 
terms  of  deviation  from  the  previous  cost  estimate.   These  changes  are 
defined  as  cost  variances.   The  sum  of  all  the  cost  variances  and  the  DE 
is  defined  as  the  Current  Estimate  (CE) . 

Each  cost  variance  listed  in  the  SAR  is  categorized  according  to 
the  cause  of  change.   These  categories  are  explicitly  defined  in 

Department  of  Defense  Manual  5000. 2M. 

a.  Quantity  Change 

A  cost  variance  is  classified  as  a  quantity  change  if  it  is 

due  to  a  change  in  the  number  of  units  of  an  end  item  of  equipment . 

b.  Schedule   Change 

A  cost  variance  is  classified  as  a  schedule  change  if  it  is 
due  to  a  change  in  procurement  or  delivery  schedule,  completion  date,  or 

intermediate  milestone  for  development  or  production. 

c.  Engineering  Change 

A  cost  variance  is  classified  as  an  engineering  change  if  it 
is  due  to  an  alteration  in  the  physical  or  function  characteristics  of  a 
system  or  item  delivered,  to  be  delivered,  or  under  development,  after 

establishment  of  such  characteristics. 

d.  Economic   Change 

A  cost  variance  is  classified  as  an  economic  change  if  it  is 

due  to  price  level  changes  in  the  economy. 

e.  Estimating  Change 

A  cost  variance  is  classified  as  an  estimating  change  if  it 
is  for  correction  of  an  error,  refinement  of  a  prior  Current  Estimate, 
or  a  change  in  program  or  cost  estimating  assumptions  and  techniques. 


f.  Support   Change 

A  cost  variance  is  classified  as  a  support  change  if  it  is 
due  to  a  change  in  cost,  regardless  of  reason,  associated  with  any  work 
breakdown  structure  element  not  included  in  the  flyaway,  rollaway,  or 

sailaway  cost. 

g.  Other   Change 

A  cost  variance  is  classified  in  the  other  change  category 
if  it  cannot  be  classified  into  one  of  the  above  categories. 


III.   ISSUES  PERTAINING  TO  THE  DATA 

A.  PROCUREMENT  COSTS 

Weapon  system  costs  are  broken  down  into  two  major  cost  categories, 
Research,  Development,  Testing  and  Evaluation  (RDT&E)  costs  and 
procurement  costs.   This  study  will  focus  on  only  the  procurement  cost 
growth  of  major  weapons  systems.   RDT&E  cost  growth  is  difficult  to 
quantify  because  of  constantly  changing  requirements  and  modifications 
in  the  initial  design  of  the  system.   These  changes,  which  occur   prior 
to  Milestone  II,  could  greatly  affect  the  RDT&E  cost  growth. 

Also,  the  major  focus  of  this  research  is  to  identify  managerial  and 
cost  estimation  problems  within  the  Department  of  Defense  after  a  system 
enters  EMD.   Analysis  of  RDT&E  cost  growth  would  be  more  beneficial  in 
defining  areas  that  need  attention  within  the  initial  stages  of  concept 
exploration  and  definition. 

B.  NEW  CATEGORICAL  DEFINITIONS 

The  criteria  for  classifying  cost  variances  into  the  SAR  categories 
has  changed  over  the  years.   This  makes  any  cross  system  analysis  of 
limited  validity.   Also,  the  SAR  categories  do  not  allow  for  easy 
identification  of  the  root  causes  behind  cost  growth  of  a  particular 
weapon  system  [Ref .  4] .   For  example,  a  cost  variance  in  the  support 
category  does  not  show  whether  the  cost  variance  was  due  to  a  decision 
to  change  the  support  requirements  or  if  a  mistake  was  made  in  initial 
estimate  of  the  support  costs.   Therefore,  the  cost  variances  need  to  be 
reclassified  into  a  new  categorization  format  before  any  analysis  of  the 
reasons  for  cost  growth  can  be  done. 

The  new  categorization  system  used  is  under  development  by  the 
Department  of  Defense  (DoD) .   The  new  categories  used  to  classify  cost 
variances  must  differentiate  between  decisions  that  are  made  that  would 


knowingly  increase  weapon  system  cost  (i.e.,  increased  range  or 

performance)  and  mistakes  that  underestimated  or  overestimated  aspects 

in  the  process  which,  if  estimated   correctly,  would  not  have  resulted 

in  cost  growth.   The  new  categories  must  also  partition  these  broad 

categories  into  smaller  subcategories  to  give  insight  into  the  most 

likely  areas  for  cost  growth  to  occur. 

The  new  categories  consist  of  two  major  categories  defined  as  to 

whether  the  cost  variance  was  due  to  a  decision  or  a  mistake.   The  cost 

variance  is  further  classified  into  one  of  six  subcategories  according 

to  the  specific  area  that  is  related. 
1.   Decision  Variance 

A  decision  variance  is  a  cost  variance  due  to  a  decision 

external  to  a  program's  defined  Milestone  II  baseline.   Examples  include 

changes  in  system  capability  and  acquisition  strategy  changes  (such  as 

dual  source  procurement,  multiyear  procurement,  etc.)  not  dictated  by 

fact  of  life  conditions.   Decision  variances  are  subcategorized  by  the 

type  of  change .   These  subcategories  are : 

a.  Dcrv 

A  cost  variance  is  classified  in  the  Dcrv  category  if  it 
results  from  a  decision  that  changes  the  system  requirements  or  results 

in  a  new  variant  of  the  system. 

b .  Dsmmi 

A  cost  variance  is  classified  in  the  Dsmmi  category  if  it 
results  from  a  decision  that  causes  changes  to  the  procurement  schedule, 
shifts  in  the  multiyear  procurement  rate  or  in  different  management 

initiatives . 

c.  Dils 

A  cost  variance  is  classified  in  the  Dils  category  if  it 
results  from  a  decision  to  change  the  Integrated  Logistical  Support 
(ILS)  factors  or  changes  in  spares  or  support  requirements. 


d .  Depf 

A  cost  variance  is  classified  in  the  Depf  category  if  it 
results  from  changes  to  the  external  program  factors  (foreign  military 

sales  (FMS) ,  labor  strikes,  etc) . 

e.  Descl 

A  cost  variance  is  classified  in  the  Descl  category  if  it 
results  from  a  decision  to  change  the  escalation  (inflationary) 

requirements . 

f.  Dother 

This  category  is  designed  to  account  for  minor  or  unforeseen 

decision  variances  that  cannot  be  categorized  into  one  of  the  above 

categories . 

2.   Mistake  Variance 

A  mistake  variance  is  a  cost  variance  not  attributable  to  post- 
Milestone  II  decisions.   Examples  include  method  errors,  omissions, 
schedule  slips  attributable  to  technical  problems,  weight  growth,  and 
inadequately  scoped  engineering  and  software  development  efforts.   An 
important  point  is  that  this  category  is  considerably  broader  than 
estimating  error:  "mistakes"  include  many  factors  that  are  manifestly 
not  considered  to  be  related  to  "cost  estimating"  (e.g.,  weight  growth). 
This  category  is  intended  to  capture  all  the  variance  that  the 
acquisition  system  as  a  whole  should  have  anticipated,  not  just  the  cost 

estimator.   Mistake  variances  are  subcategorized  by  the  type  of  change. 

a .  Mcep 

A  cost  variance  is  classified  in  the  Mcep  category  if  it  is 

due  to  a  mistake  in  estimating  the  production  costs  of  the  system. 

b.  Mcede 

A  cost  variance  is  classified  in  the  Mcede  category  if  it  is 
due  to  a  mistake  in  estimating  the  developmental  and  engineering  costs 
of  the  system. 
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c.  Mils 

A  cost  variance  is  classified  in  the  Mils  category  if  it  is 
due  to  a  mistake  in  estimating  ILS  factors,  spares  or  support 

requirements . 

d.  Msszaf 

A  cost  variance  is  classified  in  the  Mssmf  category  if  it  is 
due  to  schedule  slips  and  or  changes  in  the  management  factors  that  are 

not  attributable  to  a  decision. 

e.  Mescl 

A  cost  variance  is  classified  in  the  Mescl  category  if  it  is 

caused  by  a  mistake  in  estimating  the  escalation  requirements. 

f.  Mother 

This  category  is  designed  to  account  for  minor  or  unforeseen 
mistake  variances  that  cannot  be  categorized  into  one  of  the  above 
categories . 

This  new  classification  system  will  allow  for  a  more  detailed 
evaluation  of  the  factors  affecting  cost  growth.   It  will  indicate  the 
specific  areas  in  which  cost  growth  is  occurring  and  where  measures  to 
control  cost  growth  should  be  focused. 

C.   USING  SAR  DATA 

Each  service  was  tasked  by  the  DoD  to  reconstruct  the  data  into  the 

new  categories  using  historical  SAR  data. 
1.   Advantages  of  SAR  Data 

A  majority  of  major  weapons  systems  are  required  to  have  a 

SAR  filed  on  them.   This  allows  for  a  comprehensive  analysis  of  cost 

growth  within  the  Department  of  Defense.   Most  SARs  are  unclassified 

and  are  publicly  available.   This  allows  wide  dissemination  of  data 

and  results.   This  allows  objective  research  outside  the  government 

establishment  to  review  the  progress  on  the  control  of  cost  growth. 

Failure  to  analyze  classified  systems  should  not  significantly  impact 

the  results  of  a  cost  growth  study  since  there  are  a  limited  number  of 
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these  systems.   Furthermore,  subsequent  analyses  by  the  Office  of  the 
Secretary  of  Defense  of  all  weapons  systems  when  they  become  available, 
including  classified  systems,  will  alleviate  this  concern. 

Each  cost  variance  must  be  reported  on  the  SAR  along  with  the 
cause  of  the  variance.   This  allows  for  a  full  accounting  by  the 
services  of  these  variances.   A  minimal  amount  of  cross  referencing  will 
be  required  by  the  services  to  obtain  the  root  cause  of  the  cost 
variance  which  is  needed  to  accurately  analyze  cost  growth  [Ref  4] . 

These  advantages  make  the  use  of  SAR  data  a  necessity  once  it  is 

appropriately  modified  by  the  services  into  a  more  useful  categorization 

of  the  variances . 

2 .   Disadvantages  of  SAR  Data 

Since  the  old  SAR  categories  were  not  consistently  applied, 
obtaining  the  data  in  a  consistent  categorization  will  require  the 
cooperation  of  the  services  to  reclassify  the  data  into  the  new 
categories .   Each  service  will  also  need  to  reconstruct  the  actual  cause 
of  each  variance.   Service  cooperation  is  available,  although  it  will 
take  two  years  to  obtain  all  weapons  systems  covered  by  SAR  reporting 
requirements . 

The  categorization  of  the  cost  variances  is  a  judgment  call. 
The  decisions  by  individual  services  may  generate  associated  errors  that 
cannot  be  objectively  described. 

These  disadvantages,  while  important,  do  not  constitute  a  large 
problem.   The  errors  imparted  into  the  data  should  not  be  of  a 
sufficiently  large  magnitude  to  nullify  the  findings.   The  additional 
time  to  categorize  the  cost  variances  should  be  offset  by  the  time  saved 
in  using  the  SARs  as  described  above. 

D.   MILESTONE  II  BASELINE 

This  baseline  cost  estimate  is  considered  valid  in  the  Department  of 
Defense  since  this  is  where  the  Developmental  Estimate  (DE)  originates. 
Prior  to  this  point,  numerous  alternatives  combined  with  unproven 
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concepts  would  tend  to  introduce  gross  margins  of  error  into  any 
estimate . 

A  major  problem  when  analyzing  weapons  system  cost  growth  is  that 
the  DE  baseline  can  change.   The  baseline  is  allowed  to  change  if  major 
modifications  or  changes  to  the  weapon  system  are  performed.   This  could 
suppress  the  actual  cost  growth  of  the  system  if  the  cost  variances  are 
compared  to  the  new  baseline.   The  reconstruction  of  the  data  into  the 
new  format  by  the  services  maintains  the  Milestone  II  baseline  as  its 
initial  estimate  and  reports  all  cost  variances  from  this  point.   This 
should  allow  for  a  focus  on  all  forms  of  cost  growth  in  the  weapons 
system. 

E.  ESCALATION  ADJUSTMENT 

The  data  used  is  in  constant  base  year  dollars.   The  base  year  for 
analysis  of  cost  growth  of  those  systems  that  are  beyond  Milestone  II  is 
generally  chosen  as  the  year  the  project  enters  EMD.   Using  constant 
base  year  dollars  is  necessary  when  measuring  program  management  effects 
since  this  allows  for  consistent  comparisons  to  the  base  estimate  [Ref 
5] .   If  then  year  (budgetary)  dollars  were  used,  any  analysis  that 
combines  the  cost  variances  across  time  could  not  be  performed  since 
this  would  result  in  attempting  to  combine  different  year  dollars  which 
cannot  be  done.   Therefore,  for  this  analysis,  constant  base  year 
dollars  are  required. 

F.  QUANTITY  ADJUSTMENT 

Quantity  changes,  whether  a  change  in  the  total  quantity  procured  or 
a  change  in  the  procurement  rate,  will  greatly  affect  the  total  cost 
growth  of  a  system.   Milestone  II  cost  estimates  were  based  on  the 
Milestone  II  quantity.   Any  subsequent  change  in  quantity  could  not  have 
been  foreseen  at  Milestone  II.   Therefore,  an  adjustment  to  the  cost 
variances  needs  to  be  made  to  account  for  these  changes  in  quantity. 
This  is  defined  as  quantity  normalization. 
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1.   Quantity  Related  Cost  Equation 

Before  developing  the  quantity  normalization  equation,  it  is 
necessary  to  understand  the  composition  of  the  current  estimate  (CE)  and 
how  it  relates  to  the  Milestone  II  decision  estimate  (DE) . 

Two  types  of  costs  exist  in  the  DE .   One  type  is  cost  not 
related  to  quantity.   These  are  costs  that  do  not  change  with  changes  in 
the  quantity  to  be  procured  or  changes  in  the  procurement  rate. 
Examples  of  these  costs  include  initial  design  engineering  and  tooling. 
The  total  nonquantity- related  cost  at  the  DE  is  defined  as  N. 

The  other  type  of  cost  in  the  DE  is  quantity  related.   The  sum 
of  the  quantity-related  costs  at  Milestone  II  is  defined  as  R.   These 
change  with  changes  in  quantity  or  changes  in  the  procurement  rate. 
Examples  of  these  costs  include  material  and  labor.   The  total  quantity- 
related  cost  at  Milestone  II  is  formed  by  the  product  of  the  Milestone 
II  unit  cost  (U)  times  the  Milestone  II  quantity  to  be  procured  (Q) .   By 
evaluating  the  reasons  given  in  the  SAR  for  each  of  the  cost  variances, 
any  cost  variance  listed  in  the  SAR  can  be  classified  as  being  either 
quantity  related  or  not.   Therefore,  the  Milestone  II  DE  is  written  as: 
DE  =  N  +  R 
=  N  +  UQ 

The  total  change  in  the  nonquantity  related  costs  from  the 
decision  estimate  to  the  current  estimate  (AN)  can  be  calculated  by 
summing  all  of  the  cost  variances  that  are  not  quantity  related. 

The  total  change  in  the  quantity  related  costs  from  the  decision 
estimate  to  the  current  estimate  (AR)  can  be  calculated  by  summing  all 
of  the  cost  variances  that  are  quantity  related. 

The  current  estimate  (CE)  can  be  determined  from  the  DE  and  the 
cost  variances  by: 

CE  =  DE  +  AN  +  AR 
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A  quantity  related  cost  variance  can  be  caused  by  a  change  in 

the  quantity  to  be  procured  (AQ) ,  a  change  in  the  unit  cost  of  the  item 

(AU) ,  or  both.   Therefore,  CE  can  be  rewritten  as: 

CE  =  DE  +  AN  +  AR 

=  N  +  AN  +  R  +  AR 

=  N  +  AN  +  (U  +  AU) (Q  +  AQ) 

=  N  +  AN  +  UQ  +  QAU  +  UAQ  +  AUAQ 

By  further  analyzing  the  reasons  for  the  cost  variances  listed 

in  the  SAR,  those  cost  variances  resulting  from  a  change  in  quantity 

with  no  change  in  unit  cost  can  be  identified.   The  sum  of  these  cost 

variances  is  defined  as  UAQ.   The  remaining  quantity  related  cost 

variances  are  due  to  changes  in  quantity  with  a  corresponding  change  in 

unit  cost  or  are  due  to  a  change  in  unit  cost  with  no  change  in  the 

quantity.   The  sum  of  these  cost  variances  is  defined  as  AU(Q  +  AQ) . 

Therefore,  CE  can  be  rewritten  as: 

CE  =  N  +  AN  +  UQ  +  QAU  +  UAQ  +  AUAQ 

=  (N  +  AN)  +  UQ  +  AU(Q  +  AQ)  +  UAQ 

Define  AP  as  the  difference  between  the  current  estimate  and  the 

decision  estimate: 

AP  =  CE  -  DE 

=  (N  +  AN)  +  UQ  +  AU(Q  +  AQ)  +  UAQ     -  (N  +  UQ) 

=  AN  +  AU(Q  +  AQ)  +  AUAQ 

with  AN,  AU(Q  +  AQ) ,  and  UAQ  each  being  a  determinable  quantity. 
2.   Quantity  Normalization  Equation 

The  quantity  normalization  equation  used  in  this  research  is 
currently  under  development  by  the  Office  of  the  Secretary  of  Defense. 

The  preferred  method  of  quantity  normalization  varies  widely 
[Refs.  4,  5] .   Two  types  of  effects  due  to  quantity  must  be  addressed. 
The  first  is  that  the  final  quantity  is  changed  with  no  change  in  the 
procurement  rate.   This  study  deals  with  the  total  costs  of  the  system 
based  on  the  quantity  determined  at  Milestone  II  and  therefore  requires 
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that  this  type  of  quantity  change  not  result  in  a  cost  variance.   The 
second  type  of  quantity  effect  is  a  change  in  the  procurement  rate  with 
no  change  in  the  final  quantity.   Since  contractors  face  both  fixed  and 
variable  costs,  a  change  in  the  procurement  rate  such  as  a  schedule 
slippage  will  result  in  a  higher  unit  cost  since  additional  fixed  costs 
will  be  incurred.   This  type  of  cost  variance  must  be  fully  accounted 
for. 

The  data  collected  is  classified  as  to  whether  or  not  it  was 
quantity  related.   No  adjustment  for  quantity  is  required  if  it  is  not 
quantity  related.   The  quantity  adjustment  to  the  cost  variance  must  be 
made  to  the  base-year  adjusted  cost  variance  to  remove  any  inflationary 
effects . 

AP  is  not  quantity  normalized  since  a  change  in  the  quantity 
(AQ)  with  no  change  in  the  procurement  rate  (i.e.,  AU  =  0)  will  result 
in  a  change  in  AP. 

The  required  form  for  a  quantity  normalized  procurement  cost  C 
must  remove  any  effect  for  a  change  in  quantity  without  a  change  in 
procurement  rate  since  this  form  of  cost  variance  could  not  have  been 
foreseen  by  the  Milestone  II  estimators.   However,  C  must  fully  account 
for  procurement  rate  changes  since  these  are  attributable  to  decisions 
or  mistakes  that  should  have  been  foreseen  at  Milestone  II.   Therefore 
the  proper  form  for  C  is : 

C  =  (N  +  AN)  +  (U  +  AU)Q 

This  satisfies  the  requirements  listed  above,  namely  that 
changes  in  quantity  with  no  change  in  procurement  cost  are  ignored  but 
that  changes  in  procurement  rate  which  affect  the  unit  cost  are 
completely  accounted  for. 

The  quantity  normalized  cost  variance  AC  is  the  quantity 
normalized  CE  (C)  minus  the  DE .   This  yields: 
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AC  =    C  -  DE 

=  (N  +  AN)  +  (U  +  AU)Q  -  (N  +  UQ) 
=  AN  +  QAU 
The  equation  for  determining  AC  can  be  obtained  as  follows: 
AC  =  AN  +  QAU 

=  AN  +  QAU(Q  +  AQ) / (Q  +  AQ) 
The  terms  Q  and  AQ  can  be  taken  directly  from  the  SAR.  The 
remaining  terms,  AN  and  AU(Q  +  AQ)  can  be  determined  by  the  method 
listed  previously  when  determining  AP.   Therefore,  AC  is  the  quantity 
normalized  procurement  cost  variance  in  base  year  dollars.   AC  will  be 
used  in  the  analysis  as  the  quantity  normalized  cost  variance  to 
determine  the  cost  growth  of  the  weapon  system. 
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IV .   DATA 

Data  from  nine  weapon  systems  are  currently  available.   In  several 
years,  data  for  all  systems  reported  under  the  SAR  guidelines  will  be 
collected  and  all  of  the  data  will  be  analyzed  by  the  Office  of  the 
Secretary  of  Defense. 

Only  nine  systems  were  chosen  due  to  the  limited  time  available  for 
data  collection  for  this  study.   The  Department  of  Defense  selected  the 
first  nine  systems  so  that  they  reflect  a  representative  sample  of  the 
systems  in  each  service.   They  include  ships,  missiles  and  electronic 
systems.   No  aircraft  systems  were  available,  and  it  should  be  noted 
that  the  results  of  this  analysis  should  not  be  applied  to  any  analysis 
of  aircraft  systems .   There  are  approximately  equal  numbers  of  Army  and 
Navy  systems.   No  Air  Force  systems  were  available. 

The  systems  are  at  different  times  in  their  procurement  lifetime  and 
have  a  wide  range  of  total  costs  to  reflect  the  actual  nature  of  the 
overall  procurement  process  within  the  Department  of  Defense. 

Four  of  the  systems  were  reported  by  the  Army.  They  were  the  ATACM 
missile,  the  MLRS  rocket  system,  the  SINCGARS  electronic  system  and  the 
MSE  electronic  system.  The  remaining  five  systems  were  reported  by  the 
Navy.  These  are  the  CH\MH-53  helicopter,  the  Trident  II  submarine,  the 
DDG-51  surface  ship,  the  HARM  missile  and  the  SM-2  missile. 

The  data  for  each  weapon  system  was  constructed  from  existing  SAR 
data  by  the  respective  service.   The  data  consists  of  all  cost  variances 
reported  in  constant  base  year  dollars  from  the  SAR  since  Milestone  II 
along  with  the  SAR  categorization,  the  new  categorization,  and  whether 
or  not  the  cost  variance  is  quantity  related.   The  reported  data  also 
included  the  Milestone  II  RDT&E  and  procurement  costs  as  well  as  any 
amplifying  remarks  necessary  to  further  explain  any  cost  variances. 


18 


A.  COST  GROWTH  RATIO 

The  cost  growth  ratio  (CGR)  is  defined  as  the  ratio  of  the  quantity 
and  escalation  adjusted  cost  variances  and  the  Milestone  II  total  system 
costs  including  RDT&E  costs: 

CGR  =  AC/DE. 

The  total  system  cost  is  used  so  that  the  CGR  represents  the  change 
in  the  total  cost  of  the  system  as  opposed  to  a  percentage  change  in  the 
procurement  cost  of  the  system.   All  cost  variances  for  each  system  were 
converted  to  CGRs .   The  CGR  is  used  in  nearly  all  research  on  cost 
growth  to  show  the  magnitude  of  the  change  compared  to  the  original 
estimate.   For  instance,  a  fifty  million  dollar  change  in  a  system  that 
cost  two  hundred  million  dollars  is  much  more  significant  than  a  fifty 
million  dollar  change  in  a  ten  billion  dollar  system  since  the  first 
constitutes  a  relatively  large  change  in  the  accuracy  of  the  original 
estimate  (CGR  =  0.25),  while  the  latter  is  well  within  any  reasonable 
expectations  for  margin  of  estimating  error  (CGR  =  0.005)  . 

For  each  system,  the  CGRs  associated  with  a  particular  category, 
using  both  the  SAR  and  the  new  categorization  systems,  were  aggregated 
to  form  the  cost  growth  for  a  system  classified  by  category.   The  CGRs 
that  are  associated  with  the  mistake  categories  were  aggregated  to  form 
the  mistake  cost  growth  and  in  a  similar  manner  the  decision  categories 
were  combined  to  form  the  decision  cost  growth. 

The  sum  of  the  decision  and  mistake  cost  growths  form  the  total  cost 
growth  of  a  system  since  Milestone  II. 

B.  DOLLAR  WEIGHTED  AVERAGE 

When  systems  are  combined  into  sets  for  analysis  of  the  effects  of 
common  factors,  the  dollar  weighted  average  is  used  as  the  mean  value 
instead  of  the  arithmetic  mean.   This  is  done  to  account  for  the  greater 
impact  that  cost  growth  of  more  expensive  weapon  systems  has  on 
budgetary  considerations.   The  objective  of  this  research  is  to 
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determine  what  factors  are  causing  cost  growth  within  the  entire 
Department  of  Defense  rather  than  on  a  particular  system.  This  type  of 
analysis  should  aid  in  determining  the  impact  of  cost  growth  on  the 
military's  procurement  budget.   A  strictly  arithmetic  average  is  not 
appropriate  in  a  budgetary  context  since  the  effect  of  less  expensive 
systems  would  skew  the  combined  CGRs  away  from  the  expected  cost  growth 
in  the  procurement  of  all  weapons  systems. 

The  weapon  systems  will  be  divided  into  various  sets  in  the  analysis 
of  their  cost  growth  (e.g.,  Army  and  Navy  systems) .  The  dollar  weighted 
average  (DWA)  for  a  set  of  weapon  systems  is  determined  by: 

DWA  =  Z^w^CGRj        i  e  {all  systems  in  the  set} 
where   wL   =  TCt/    (EiTCJ      Vi  e  {all  systems  in  the  set} 
EiWi  =1  i  e  {all  systems  in  the  set} 

TCi  =  Milestone  II  total  system  cost  of  system  i. 
The  TC^s  for  the  weapon  systems  in  the  set  must  all  be  converted  to  the 
same  year  dollars.   For  this  analysis,  all  Milestone  II  estimates  were 
converted  to  1993  constant   dollars  using  the  OSD/NAVCOMPT  Guidance  of 
March  1993  dated  26  March  1993. 

C.  SINCGARS  SYSTEM  ANOMALY  AND  TREATMENT 

The  Army's  SINCGARS  electronic  system  has  a  CGR  of 
-0.777  for  the  seventh  year  since  Milestone  II.   This  results  from  a 
decision  in  that  year  for  a  large  scale  reduction  in  the  system.   This 
decision  is  not  typical  of  most  weapons  systems.   The  data  for  the  first 
six  years  is  typical  and  will  be  used.   The  first  six  years  since 
Milestone  II  had  a  total  CGR  of  0.105. 

D.  TOTAL  DATA  SET 

Table  1  displays  the  Milestone  II  year,  the  Milestone  II  decision 
estimate  (DE)  in  millions  of  base  year  dollars,  and  the  number  of  years 
of  data  since  Milestone  II  available.       Each  data  series  extends 
only  through  the  final  year  reported  by  each  service. 
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Table  2  displays  the  unweighted  cost  growth  ratios  for  the  decision 
categories  and  the  total  decision  cost  growth  ratio  for  each  system.   It 
also  displays  the  dollar  weighted  average  of  each  category. 

Table  3  displays  the  unweighted  cost  growth  ratios  for  the  mistake 
categories  and  the  total  mistake  cost  growth  ratio  for  each  system.   It 
also  displays  the  dollar  weighted  average  of  each  category. 

Table  4  displays  the  unweighted  decision,  mistake,  and  total  cost 
growth  ratios  for  each  system  and  the  dollar  weighted  average  of  each 
category. 

TABLE  1.   BASELINE  SYSTEM  DATA 


System 

Milestone  II  Year 

DE  (in  Millions  of 
Base  Year  Dollars) 

Years  of 
Data  Used 

ATACM 

1990 

1506.0 

2 

CH\MH-53 

1979 

510.0 

12 

DDG-51 

1987 

16723 .8 

4 

HARM 

1982 

3274.3 

10 

MLRS 

1984 

2077. 8 

7 

MSE 

1989 

3548.4 

3 

SINCGARS 

1983 

4145.0 

6 

SM-2 

1984 

6571.5 

8 

Trident 

1986 

14258.9 

4 

TABLE  2 


UNWEIGHTED  CGRS  FOR  DECISION  CATEGORIES 


Dcrv 

Dsmmi 

Dils 

Depf 

Descl 

Dother 

Decision 

ATACM 

0 

0.023 

0 

0.015 

0 

0 

0.038 

CH\MH-53 

-0.009 

0.018 

-0.310 

0 

0 

0 

-0.301 

DDG-51 

0.041 

0.100 

0.009 

0 

0.006 

0 

0.156 

HARM 

-0.142 

0.258 

0.015 

0.030 

0 

0 

0.161 

MLRS 

0.079 

0.014 

0.035 

-0.008 

0 

0 

0.120 

MSE 

0 

0.018 

0 

0 

0 

0 

0.018 

SINCGARS 

0.156 

0 

-0.037 

0 

0 

0 

0.119 

SM-2 

0.016 

0.072 

0.010 

-0.025 

0 

0 

0.073 

Trident 

-0.022 

0 

0 

0 

0 

0 

-0.022 

Average 

0.015 

0.061 

-0.001 

-0.001 

0.002 

0 

0.075 
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TABLE  3 


UNWEIGHTED  CGRS  FOR  MISTAKE  CATEGORIES 


Mcep 

Mcede 

Mils 

Mssmf 

Mescl 

Mother 

Mistake 

ATACM 

0 

0 

-0.011 

0 

0  .  020 

0 

0.C09 

CH\MH-53 

0.529 

0 

0.047 

0 

0 

0 

0.576 

DDG-51 

0.113 

0 

-0.005 

0 

0 

0 

0  .108 

HARM 

0.037 

0 

0.022 

-0.005 

0.002 

0 

0.056 

MLRS 

-0.036 

0 

-0.006 

0 

0 

-0.020 

-0.062 

MSE 

0.082 

-0.010 

0 

0.003 

0 

0 

0  .075 

SINCGARS 

-0.030 

0 

0.005 

0.011 

0 

0 

-0.014 

SM-2 

-0.033 

0 

-0.009 

0 

-0. 014 

0 

-0.056 

Trident 

-0. 039 

-0.001 

0 

0 

0  .  014 

0  .002 

-0. 024 

Average 

0.  031 

-0. 001 

-0.001 

0.  001 

0.003 

0 

0.038 

TABLE  4 .   UNWEIGHTED  SYSTEM  COST  GROWTH  RATIOS 

Decision 

Mistake 

Total 

ATACM 

0.038 

0.009 

0  .  047 

CH\MH-53 

-0.301 

0.576 

0.276 

DDG-51 

0.156 

0.108 

0.264 

HARM 

0.161 

0.056 

0.217 

MLRS 

0.120 

-0.062 

0.058 

MSE 

0.018 

0.075 

0.092 

SINCGARS 

0.119 

-0.014 

0.105 

SM-2 

0.073 

-0.056 

0.017 

Trident 

-0.022 

-0.024 

-0.046 

Average 

0.075 

0.033 

0.  108 
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V.   METHODOLOGY  AND  RESULTS 

A.   USEFULNESS  OF  NEW  CATEGORIES 

In  order  to  more  readily  identify  areas  of  cost  growth,  it  is 
necessary  to  classify  the  cost  variances  into  a  more  useable  and  more 
detailed  categorization  system.   The  new  system  should  classify  the  cost 
growths  into  more  categories  than  the  SAR  categorization  method. 

This  system  shows  data  counts,  and  it  would  be  inappropriate  to  use 
analysis  of  variance  (ANOVA)  since  the  data  is  not  normally  distributed. 
However,  a  simple  comparison  of  the  number  of  categories  needed  under 
each  system  will  suffice  in  this  determination. 

Table  5  displays  the  number  of  SAR  categories  as  compared  to  the 
number  of  new  categories  used  to  classify  each  system' s  cost  variances 
since  Milestone  II. 

As  can  be  seen  from  the  table,  all  systems  require  a  larger  number 
of  categories  to  classify  the  cost  variances  using  the  new  system  than 
using  the  SAR  system.   Therefore,  those  factors  having  the  greatest 
impact  on  cost  growth  can  be  more  readily  identified  and  the  underlying 
causes  of  each  of  the  most  significant  contributors  can  be  determined. 
TABLE  5.   NUMBER  OF  REQUIRED  CATEGORIES 


System 

SAR 
Categories 

New 
Categories 

ATACM 

3 

4 

CH\MH-53 

4 

5 

DDG-51 

4 

6 

HARM 

4 

8 

MLRS 

2 

7 

MSE 

3 

4 

SINCGARS 

4 

5 

SM-2 

4 

7 

Trident  II 

2 

5 
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B.   STATISTICAL  APPROACH 

1.  Regression  And  Time  Series  Models 

A  regression  model  of  the  cost  growth  ratio  would  appear  to  be 
useful  when  analyzing  the  cost  growth  of  weapon  systems.   However,  with 
the  limited  data  available,  combined  with  the  large  variation  in  CGRs , 
the  only  regression  model  that  fits  is  the  full  or  saturated  model  which 
does  not  provide  any  beneficial  information.   Also,  since  the  objective 
of  this  research  is  not  to  predict  the  cost  growth  of  a  specific  system 
but  instead  to  describe  the  cost  growth  of  the  entire  Department  of 
Defense,  a  regression  model  would  not  be  of  use. 

A  time  series  approach  modeling  the  cost  growth  versus  time  of 
weapon  systems  is  also  of  little  benefit.   The  uniqueness  of  an 
individual  weapon  system  and  the  decisions  that  affect  a  particular 
system  do  not  allow  for  general  system  models  of  cost  growth  for  these 

weapon  systems. 

2 .  Data  Groups 

The  weapon  systems  will  be  broken  into  sets  to  determine  if  a 
specific  attribute  of  each  set  of  systems  has  an  impact  on  the  cost 
growth  of  the  set.   The  following  groupings  or  sets  of  weapons  systems 
will  be  used  to  analyze  the  data:  low  cost  versus  high  cost  systems, 
missile  systems  versus  nonmissile  systems,  and  Army  versus  Navy  systems. 
An  analysis  of  those  categories  where  a  majority  of  the  cost  growth 
occurs  in  each  of  these  sets  can  be  used  to  evaluate  the  causes  of  cost 
growth.   The  reasons  for  each  of  the  groupings  will  be  discussed  in 

detail  in  each  of  the  sections  below. 

3 .  Testing  For  Common  Means 

The  assumption  of  normality  of  the  cost  growth  ratios  in  order 
to  test  for  common  means  using  parametric  tests  would  be  difficult  to 
make  since  the  CGR  cannot  go  below  a  value  of  -1.0.   The  appropriate 
nonparametric  test  for  common  means  is  the  Wilcoxon  Rank  Sum  Test  which 
is  equivalent  to  the  Mann-Whitney  U  Test.   The  value  of  the  Wilcoxon 
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test  statistic  (W)  will  be  given  along  with  the  associated  level  of 

significance  (p  value) . 

If  the  p  value  associated  with  the  Wilcoxon  test  is  not 

determinable  as  significant  or  not  (i.e.,  between  0.1  and  0.15)  and 

paired  system  data  is  being  evaluated,  the  Friedman  nonparametric  test 

will  be  used  to  determine  whether  or  not  to  accept  the  null  hypothesis. 

The  Friedman  chi-squared  value  (X2)  will  be  given  along  with  the 

associated  level  of  significance  (p  value)  when  this  test  is  conducted. 

[Ref  6] 

a.  Dollar   Weighting   the  Data 

The  mistake,  decision,  and  total  CGRs  of  each  element  i  in  a 

set  need  to  be  transformed  to  account  for  the  different  total  cost  each 

system  has.   This  allows  for  a  larger  weight  to  be  placed  on  the  CGR  of 

the  higher  cost  systems  in  the  set  and  a  lesser  weight  to  be  placed  on 

the  less  expensive  systems  in  the  set.   Each  CGR  will  be  adjusted  using 

a  dollar  weighting  factor.   The  dollar  weighting  factor  (wj  is 

determined  for  each  element  i  of  a  set  of  weapon  systems  and  is 

equivalent  to  the  wt  defined  in  calculating  the  dollar  weighted  average. 

Before  a  test  for  common  means  is  performed,  the  mistake,  decision,  and 

total  CGRs  of   each  element  i  in  the  set  is  multiplied  by  its  dollar 

weighting  factor  Wi-   The  dollar  weighted  CGRs  are  then  used  for  the 

common  means  tests . 

C.   DECISION  VERSUS  MISTAKE  COST  GROWTH 

An  analysis  of  the  difference  between  decision  cost  growth  and 

mistake  cost  growth  of  all  of  the  systems  is  required  to  determine  how 

accurate  Milestone  II  estimates  are  and  to  identify  those  areas  where 

improvements  in  our  estimating  techniques  need  to  be  made. 
1.   Hypothesis 

The  Wilcoxon  Rank  Sum  Test  is  used  to  test  the  null  hypothesis 

that  the  mean  cost  growth  due  to  decisions  is  equal  to  the  mean  cost 

growth  due  to  mistakes.   The  alternative  hypothesis  is  that  the  mean 
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cost  growth  for  decisions  is  greater  than  the  mean  cost  growth  due  to 

mistakes .   The  common  means  test  is  performed  on  the  dollar  weighted 

data  for  all  nine  weapon  systems. 
2 .   Data 

Table  6  displays  the  dollar  weighted  decision,  mistake  and  total 

cost  growths  for  all  the  weapon  systems. 

TABLE  6.   COST  GROWTH  RATIOS  FOR  EACH  SYSTEM 


System 

Decision 

Mistake 

Total 

ATACM 

0.001 

0.000 

0  .  001 

CH\MH-53 

-0.004 

0.008 

0.004 

DDG-51 

0.048 

0.033 

0.081 

HARM 

0.011 

0.004 

0.015 

MLRS 

0.005 

-0. 003 

0.002 

MSE 

0.001 

0.004 

0.  006 

SINCGARS 

0.010 

-0.001 

0.009 

SM-2 

0.010 

-0.007 

0.002 

Trident  II 

-0.006 

-0.006 

-0.012 

3.  Results  of  Decision  Versus  Mistake  Cost  Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision  and 
mistake  cost  growth  yields  a  test  statistic  value  of  W  =  72  with  a  p 
value  of  0.13.   This  may  or  may  not  be  significant,  so  a  Friedman  test 
was  performed  and  yields  a  test  statistic  value  of  X2    =  2.778  with  a  p 
value  of  0.096.   This  is  significant  and  the  null  hypothesis  of  equal 
means  is  rejected.   Therefore,  the  mean  decision  cost  growth  is  greater 
than  the  mean  mistake  cost  growth.   The  dollar  weighted  average  (DWA) 

decision  cost  growth  is  0.075  and  the  DWA  mistake  cost  growth  is  0.033. 

4 .  Categorical  Analysis 

Table  7  displays  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories.   No  statistical  tests  were  performed  to 
determine  which  categories  contain  the  largest  cost  growth;  only  a 
visual  inspection  is  required. 
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TABLE  7 .   AVERAGE  CATEGORY  CGRS  FOR  ALL 

SYSTEMS 

Decision 

Dcrv 

Dsmmi 

Dils 

Depf 

Descl 

Dother 

CGR 

0.015 

0.061 

-0. 001 

-0.001 

0.002 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mssmf 

Mescl 

Mother 

CGR 

0  .  031 

-0.001 

-0.001 

0.001 

0.003 

0 

The  majority  of  decision  cost  growth  occurs  in  the  Dsmmi 
category  and  has  a  dollar  weighted  average  value  of  0.061.   The  majority 
of  mistake  cost  growth  occurs  in  the  Mcep  category  and  has  a  dollar 
weighted  average  value  of  0.031. 

D.   LOW  COST  VERSUS  HIGH  COST  SYSTEMS 

A  determination  of  the  relationship  between  system  cost  and  system 
cost  growth  is  needed  to  determine  if  a  difference  exists  between  the 
cost  growth  of  low  cost  systems  and  the  cost  growth  of  high  cost 
systems.   A  low  cost  system  is  defined  as  any  system  whose  total  system 
cost  estimated  at  Milestone  II  is  less  than  three  and  a  half  billion 
dollars.   The  value  of  three  and  a  half  billion  dollars  is  somewhat 
arbitrary,  but  this  number  was  chosen  to  split  the  number  of  systems  in 
each  grouping  approximately  in  half  and  to  ensure  that  an  approximately 
equal  ratio  of  Army  to  Navy  systems  exist  in  each  set  to  eliminate  any 
possible  service  effects.   The  low  cost  systems  are  the  ATACM,  the 
CH\MH-53,  the  HARM  and  the  MLRS .   The  systems  in  the  high  cost  category 

are  the  DDG-51,  the  MSE,  the  SINCGARS,  the  SM-2  and  the  Trident  II. 

1.  Hypothesis 

The  Wilcoxon  Rank  Sum  Test  is  used  to  test  the  null  hypothesis 
that  the  mean  cost  growth  of  low  cost  systems  is  equal  to  the  mean  cost 
growth  of  high  cost  systems.   The  alternative  hypothesis  is  that  the  low 

cost  systems  have  a  higher  mean  cost  growth  than  the  high  cost  systems . 

2.  Data 

Table  8  displays  the  dollar  weighted  cost  growth  ratios  for  the 
low  cost  weapon  systems. 
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Table  9  displays  the  dollar  weighted  cost  growth  ratios  for  the 
high  cost  weapon  systems . 

TABLE  8.   DOLLAR  WEIGHTED  CGRS  FOR  LOW  COST  SYSTEMS 


System 

Decision 

MistaKe 

Total 

ATACM 

0.006 

0.001 

0.008 

CH\MH-53 

-0.029 

0.055 

0.026 

HARM 

0.075 

0.026 

0.102 

MLRS 

0.033 

-0. 017 

0  .  016 

TABLE  9 


DOLLAR  WEIGHTED  CGRS  FOR  HIGH  COST  SYSTEMS 


System 

Decision 

Mistake 

Total 

DDG-51 

0.  056 

0.039 

0.095 

MSE 

0.001 

0.005 

0.007 

SINCGARS 

0.012 

-0.001 

0.011 

SM-2 

0.011 

-0.009 

0.003 

Trident  II 

-0.007 

-0.  008 

-0.014 

3 .   Comparison  between  High  Cost  and  Low  Cost  Systems 

a.  Total  Cost  Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  total  cost 
growth  of  low  cost  and  high  cost  systems  yields  a  test  statistic  value 
of  W  =  19  with  a  p  value  of  0.08.   This  is  significant  and  the  null 
hypothesis  of  equal  means  is  rejected.   Therefore,  the  mean  total  cost 
growth  of  low  cost  systems  is  greater  than  the  mean  total  cost  growth  of 
high  cost  systems .   The  DWA  total  cost  growth  for  low  cost  systems  is 

0.152  and  the  DWA  total  cost  growth  for  high  cost  systems  is  0.100. 

b.  Decision   Cost   Growth   Comparison 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  growth  of  low  cost  and  high  cost  systems  yields  a  test  statistic 
value  of  W  =  22  with  a  p  value  of  0.311.   This  is  not  significant  and 
the  null  hypothesis  of  equal  means  is  not  rejected. 
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c.  Mistake   Cost   Growth   Comparison 

The  Wilcoxon  rank  sum  test  for  common  means  for  mistake  cost 
growth  of  low  cost  and  high  cost  systems  yields  a  test  statistic  value 
of  W  =  21  with  a  p  value  of  0.35.   This  is  not  significant  and  the  null 
hypothesis  of  equal  means  is  not  rejected. 

The  difference  between  the  mistake  cost  growths,  while  not 

statistically  different,  is  substantively  different.   The  total  cost 

growth  of  low  cost  systems  is  greater  than  that  of  high  cost  systems, 

and  it  can  be  observed  that  the  difference  between  the  mistake  cost 

growths  is  the  driving  factor  behind  this  difference.   The  DWA  mistake 

cost  growth  for  low  cost  systems  is  0.066,  while  the  DWA  total  cost 

growth  for  high  cost  systems  is  0.027. 
4.   Low  Cost  Systems 

a.  Decision   Versus   Mistake   Cost   Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  growth  and  mistake  cost  growth  of  low  cost  systems  yields  a  test 
statistic  value  of  W  =  17  with  a  p  value  of  0.38.   This  is  not 

significant  and  the  null  hypothesis  of  equal  means  is  not  rejected. 

b.  Categorical   Analysis 

Table  10  displays  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories  for  low  cost  systems.   No  statistical 
tests  were  performed  to  determine  which  categories  contain  the  largest 
cost  growth;  only  a  visual  inspection  is  required. 

TABLE  10.   AVERAGE  CATEGORY  CGRS  FOR  LOW  COST  SYSTEMS 


Decision 

Dcrv 

Dsmmi 

Dils 

Depf 

Descl 

Dother 

CGR 

-0.046 

0.130 

-0.013 

0.  014 

0 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mssmf 

Mescl 

Mother 

CGR 

0.058 

0 

0.011 

-0.002 

0.004 

-0.005 

The  majority  of  decision  cost  growth  for  low  cost  systems 
occurs  in  the  Dsmmi  category  and  has  a  dollar  weighted  average  value  of 
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0.13  0.   The  majority  of  mistake  cost  growth  in  low  cost  systems  occurs 

in  the  Mcep  category  and  has  a  dollar  weighted  average  value  of  0.058. 
5.   High  Cost  Systems 

a.  Decision   Versus  Mistake   Cost   Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  growth  and  mistake  cost  growth  of  high  cost  systems  yields  a  test 
statistic  value  of  W  =  22  with  a  p  value  of  0.14.   This  may  or  may  not 
be  significant,  so  a  Friedman  test  was  performed  and  yields  a  test 
statistic  value  of  X2    =  1.8  with  a  p  value  of  0.18.   This  is  not 
significant  and  the  null  hypothesis  of  equal  means  is  not  rejected. 

The  difference  between  the  mistake  and  decision  cost 
growths,  while  not  statistically  different,  is  substantively  different 
when  compared  to  the  difference  between  the  decision  and  mistake  cost 
growths  of  low  cost  systems.   The  DWA  mistake  cost  growth  for  high  cost 
systems  0.027  and  the  DWA  decision  cost  growth  for  high  cost  systems  is 

0.074. 

b.  Categorical   Analysis 

Table  11  displays  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories  for  high  cost  systems.   No  statistical 
tests  were  performed  to  determine  which  categories  contain  the  largest 
cost  growth;  only  a  visual  inspection  is  required. 

TABLE  11.   AVERAGE  CATEGORY  CGRS  FOR  HIGH  COST  SYSTEMS 


Decision 

Dcrv 

Dsmmi 

Dils 

Depf 

Descl 

Dother 

CGR 

0.026 

0.048 

0.001 

-0.004 

0.002 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mssmf 

Mescl 

Mother 

CGR 

0.026 

-0.001 

-0.003 

0.001 

0.002 

0.001 

The  majority  of  decision  cost  growth  for  high  cost  systems 
occurs  in  the  Dsmmi  category  and  has  a  dollar  weighted  average  value  of 
0.048.   The  majority  of  mistake  cost  growth  for  high  cost  systems  occurs 
in  the  Mcep  category  and  has  a  dollar  weighted  average  value  of  0.026. 
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E.   MISSILE  SYSTEMS  VERSUS  OTHER  SYSTEMS 

An  analysis  of  the  cost  growth  of  missile  systems  compared  to  other 
weapons  systems  is  beneficial  since  it   will  allow  for  identification  of 
differences  in  the  procurement  and  estimation  processes  of  these  systems 
that  may  need  to  be  evaluated  in  further  detail  in  future  studies .   The 
missile  systems  consist  of  the  ATACM,  the  HARM  and  the  SM-2.   These  two 
sets  have  approximately  equal  low  cost  to  high  cost  ratios  and  Army  to 
Navy  system  ratios.   This  should  remove  any  effects  that  system  cost 
would  have  on  the  data.   The  data  was  then  dollar  weighted  within  each 

set . 

1.  Hypothesis 

The  Wilcoxon  Rank  Sum  Test  is  performed  to  test  the  null 
hypothesis  that  the  mean  cost  growth  of  missile  systems  is  equal  to  the 
mean  cost  growth  of  other  weapon  systems.   The  alternative  hypothesis  is 
that  the  missile  systems  have  a  lower  mean  cost  growth  than  the  other 

weapon  systems . 

2 .  Data 

Table  12  displays  the  dollar  weighted  cost  growths  for  missile 
systems . 

Table  13  displays  the  dollar  weighted  cost  growths  for  the 
nonmissile  systems. 

TABLE  12 .   DOLLAR  WEIGHTED  CGRS  FOR  MISSILE  SYSTEMS 


System 

Decision 

Mistake 

Total 

ATACM 

0.004 

0.001 

0.005 

HARM 

0.050 

0.018 

0.068 

SM-2 

0.042 

-0.032 

0.010 
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TABLE  13 


DOLLAR  WEIGHTED  CGRS  FOR  NONMISSILE  SYSTEMS 


System 

Decision 

Mistake 

Total 

CH\MH-53 

-0.006 

0.011 

0.005 

DDG-51 

0.062 

0.043 

0.  104 

MSE 

0.006 

-0.003 

0.003 

SINCGARS 

0.001 

0.006 

0.007 

SM-2 

0.013 

-0.002 

0.  012 

Trident  II 

-0.008 

-0.008 

-0.016 

3.   Comparison  between  Missile  and  Nonmissile  Systems 

a.  Total  Cost  Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  total  cost 
growth  of  missile  and  nonmissile  systems  yields  a  test  statistic  value 
of  W  =  17.5  with  a  p  value  of  0.27.   This  is  not  significant  and  the 

null  hypothesis  of  equal  means  is  not  rejected. 

b.  Decision   Cost   Growth   Comparison 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  growth  of  missile  and  nonmissile  systems  yields  a  test  statistic 
value  of  W  =  19  with  a  p  value  of  0.16.   This  is  not  significant  and  the 

null  hypothesis  of  equal  means  is  not  rejected. 

c.  Mistake   Cost   Growth   Comparison 

The  Wilcoxon  rank  sum  test  for  common  means  for  mistake  cost 
growth  of  missile  and  nonmissile  systems  yields  a  test  statistic  value 
of  W  =  14  with  a  p  value  of  0.40.   This  is  not  significant  and  the  null 
hypothesis  of  equal  means  is  not  rejected. 

The  difference  between  the  mistake  cost  growths  for  missile 
and  nonmissile  systems,  while  not  statistically  different,  is 
substantively  different.   It  can  be  readily  observed  that  the  difference 
between  the  mistake  cost  growths  is  relevant  when  compared  to  the 
decision  cost  growths  as  is  discussed  below.   The  DWA  mistake  cost 
growth  for  missile  systems  is  -0.014  and  the  DWA  total  cost  growth  for 
nonmissile  systems  is  0.046. 
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4 .   Missile  Systems 

a.  Decision    Versus   Mistake    Cost    Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  growth  and  mistake  cost  growth  of  missile  systems  yields  a  test 
statistic  value  of  W  =  7  with  a  p  value  of  0.076.   This  is  significant 
and  the  null  hypothesis  of  equal  means  is  rejected.   The  DWA  mistake 
cost  growth  for  missile  systems  -0.014  and  the  DWA  decision  cost  growth 

for  missile  systems  is  0.097. 

b.  Categorical   Analysis 

Table  14  displays  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories  for  missile  systems.   No  statistical 
tests  were  performed  to  determine  which  categories  contain  the  largest 
cost  growth;  only  a  visual  inspection  is  required. 

TABLE  14 .   AVERAGE  CATEGORY  CGRS  FOR  MISSILE  SYSTEMS 


Decision 

Dcrv 

Dsmmi 

Oils 

Depf 

Descl 

Dother 

CGR 

-0.035 

0.125 

0.010 

-0.003 

0 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mssmf 

Mescl 

Mother 

CGR 

-0.007 

0 

0.001 

-0.002 

-0.005 

0 

The  majority  of  decision  cost  growth  for  missile  systems 

occurs  in  the  Dsmmi  category  and  has  a  dollar  weighted  average  value  of 

0.125.   The  majority  of  mistake  cost  growth  for  missile  systems  occurs 

in  the  Mcep  category  and  has  a  dollar  weighted  average  value  of  -0.007. 
5.   Nonmissile  Systems 

a.  Decision   Versus  Mistake   Cost   Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  growth  and  mistake  cost  growth  of  nonmissile  systems  yields  a  test 
statistic  value  of  W  =  38  with  a  p  value  of  0.43.   This  is  not 

significant  and  the  null  hypothesis  of  equal  means  is  not  rejected. 

b.  Categorical   Analysis 

Table  15  displays  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories  for  nonmissile  systems.   No  statistical 
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tests  were  performed  to  determine  which  categories  contain  the  largest 
cost  growth;  only  a  visual  inspection  is  required. 

TABLE  15.   AVERAGE  CATEGORY  CGRS  FOR  NONMISSILE  SYSTEMS 


Decision 

Dcrv 

Dsmmi 

Dils 

Depf 

Descl 

Dother 

CGR 

0.030 

0.  042 

-0.005 

0 

0.002 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mssmf 

Mescl 

Mother 

CGR 

0.042 

-0.001 

-0.001 

0.001 

0.005 

0 

The  majority  of  decision  cost  growth  for  nonmissile  systems 
occurs  in  two  categories,  the  Dsmmi  category  and  the  Dcrv  category.   The 
Dsmmi  category  has  a  dollar  weighted  average  value  of  0.042.   The  Dcrv 
category  has  a  dollar  weighted  average  value  of  0.030.   The  majority  of 
mistake  cost  growth  for  non  missile  systems  occurs  in  the  Mcep  category 
and  has  a  dollar  weighted  average  value  of  0 .  042 . 

F.   ARMY  VERSUS  NAVY  SYSTEMS 

A  comparison  of  Navy  cost  growth  versus  Army  cost  growth  is  required 
to  determine  which  service,  if  either,  has  a  better  estimation  or 

decision  making  process. 

1.  Hypothesis 

The  Wilcoxon  Rank  Sum  Test  is  performed  to  test  the  null 
hypothesis  that  the  mean  cost  growth  of  Army  systems  is  equal  to  the 
mean  cost  growth  of  Navy  systems.   The  alternative  hypothesis  is  that 

the  means  are  not  equal . 

2.  Data 

Table  16  displays  the  dollar  weighted  cost  growths  for  Army 
systems  along  with  the  dollar  weighted  average  (DWA)  of  these  systems. 

Table  17  displays  the  dollar  weighted  cost  growths  for  the  Navy 
systems  along  with  the  dollar  weighted  average  (DWA)  of  these  systems. 
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TABLE  16.   DOLLAR  WEIGHTED  CGRS  FOR  ARMY  SYSTEMS 


System 

Decision 

Mistake 

Total 

ATACM 

0.  004 

0.001 

0.005 

MLRS 

0.023 

-0. 012 

0.011 

MSE 

0.005 

0.021 

0.026 

SINCGARS 

0.048 

-0.006 

0.043 

TABLE  17 .   DOLLAR  WEIGHTED 

CGRS  FOR  NAVY  SYSTEMS 

System 

Decision 

Mistake 

Total 

CH\MH-5  3 

-0  .006 

0  .011 

0.005 

DDG-51 

0.  060 

0.042 

0  .  102 

HARM 

0.  014 

0.  005 

0.020 

SM-2 

0.  012 

-0.009 

0.003 

Trident  II 

-0. 007 

-0.008 

-0 . 016 

3.   Comparison  between  Army  and  Navy  Systems 

a.  Total    Cost   Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  total  cost 
growth  of  Army  and  Navy  systems  yields  a  test  statistic  value  of  W  = 
21.5  with  a  p  value  of  0.21.   This  is  not  significant  and  the  null 

hypothesis  of  equal  means  is  not  rejected. 

b.  Decision   Cost   Growth   Comparison 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 
cost  growth  of  Army  and  Navy  systems  yields  a  test  statistic  value  of  W 
=  22  with  a  p  value  of  0.31.   This  is  not  significant  and  the  null 

hypothesis  of  equal  means  is  not  rejected. 

c.  Mistake  Cost  Growth   Comparison 

The  Wilcoxon  rank  sum  test  for  common  means  for  mistake  cost 
growth  of  Army  and  Navy  systems  yields  a  test  statistic  value  of  W  =  18 
with  a  p  value  of  0.31.   This  is  not  significant  and  the  null  hypothesis 
of  equal  means  is  not  rejected. 
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4.   Army  Systems 

a.  Decision   Versus  Mistake   Cost   Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 

cost  growth  and  mistake  cost  growth  of  Army  systems  yields  a  test 

statistic  value  of  W  =  12  with  a  p  value  of  0.045.   This  is  significant 

and  the  null  hypothesis  of  equal  means  is  rejected.   The  DWA  mistake 

cost  growth  for  Army  systems  0.004  and  the  DWA  decision  cost  growth  for 

Army  systems  is  0.081. 

jb.  Categorical   Analysis 

Table  18  displays  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories  for  Army  systems.   No  statistical  tests 
were  performed  to  determine  which  categories  contain  the  largest  cost 
growth;  only  a  visual  inspection  is  required. 

TABLE  18.   AVERAGE  CATEGORY  CGRS  FOR  ARMY  SYSTEMS 


Decision 

Dcrv 

Dsmmi 

Dils 

Depf 

Descl 

Dother 

CGR 

0.079 

0.011 

-0.008 

0 

0 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mssmf 

Mescl 

Mother 

CGR 

0.004 

-0.003 

0 

0.005 

0.002 

-0.004 

The  majority  of  decision  cost  growth  in  Army  systems  occurs 

in  the  Dcrv  category  and  has  a  dollar  weighted  average  value  of  0.079. 

The  majority  of  mistake  cost  growth  in  Army  systems  occurs  in  two 

categories,  the  Mcep  category  and  the  Mssmf  category.   The  Mcep  category 

has  a  dollar  weighted  average  value  of  0.004.   The  Mssmf  category  has  a 

dollar  weighted  average  value  of  0.005. 
5.   Navy  Systems 

a.  Decision   Versus  Mistake  Cost   Growth 

The  Wilcoxon  rank  sum  test  for  common  means  for  decision 

cost  growth  and  mistake  cost  growth  of  Navy  systems  yields  a  test 

statistic  value  of  W  =  23  with  a  p  value  of  0.18.   This  is  not 

significant  and  the  null  hypothesis  of  equal  means  is  not  rejected. 
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b.       Categorical   Analysis 

Table  19  displays  the  dollar  weighted  average  CGRs  of  the 
decision  and  mistake  categories  for  Navy  systems.   No  statistical  tests 
were  performed  to  determine  which  categories  contain  the  largest  cost 
growth;  only  a  visual  inspection  is  required. 

TABLE  19.   AVERAGE  CATEGORY  CGRS  FOR  NAVY  SYSTEMS 


Decision 

Dcrv 

Dsmmi 

Dils 

Depf 

Descl 

Dother 

CGR 

-0. 002 

0.074 

0  .  001 

-0.001 

0.002 

0 

Mistake 

Mcep 

Mcede 

Mils 

Mssmf 

Mescl 

Mother 

CGR 

0.038 

0 

-0.001 

0 

0.003 

0.  001 

The  majority  of  decision  cost  growth  in  Navy  systems  occurs 
in  the  Dsmmi  category  and  has  a  dollar  weighted  average  value  of  0.074. 
The  majority  of  mistake  cost  growth  in  Navy  systems  occurs  in  the  Mcep 
category  and  has  a  dollar  weighted  average  value  of  0.038. 
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VI .   CONCLUSIONS 

A.  NEW  CATEGORIES 

The  new  categorization  systems  breaks  out  the  cost  growth  into  more 
categories  than  the  SAR  system.   This  system  is  better  at  detecting 
sources  of  cost  growth  in  a  particular  weapon  system  and  enabling 
correction  of  that  source.   It  also  shows  when  mistakes  or  decisions  are 
the  major  effect  behind  the  cost  growth  of  a  system. 

B.  DOLLAR  WEIGHTED  AVERAGES 

Using  dollar  weighted  averages  adjusts  the  findings  to  account  for 
the  greater  effect  more  expensive  weapon  systems  have  on  budgetary 
considerations.   This  allows  for  a  more  realistic  interpretation  of  the 
cost  growth  and  its  effect  on  the  defense  budget.   If  an  arithmetic 
average  is  used,  the  data  is  equally  weighted,  giving  the  cost  growth  of 
low  cost  systems  an  equal  weight  on  the  overall  cost  growth.   This  would 
tend  to  skew  the  results  and  not  allow  them  to  be  applied  directly  to 
the  entire  military  budget. 

An  example  of  this  effect  occurs  when  evaluating  the  decision  and 
mistake  cost  growth  ratios  of  all  the  weapon  systems.   If  an  arithmetic 
average  is  used,  the  average  decision  cost  growth  ratio  is  0.040  and  the 
average  mistake  cost  growth  ratio  is  0.074.   This  is  opposite  to  the 
findings  using  dollar  weighted  averages,  where  the  decision  cost  growth 
ratio  is  0.075  and  the  mistake  cost  growth  ratio  is  0.033.   By  using  an 
arithmetic  mean,  it  would  lead  one  to  conclude  that  7.4%  mistake  cost 
growth  is  occurring  in  the  defense  budget,  when  in  reality,  only  3.3%  is 
occurring.   Therefore,  dollar  weighted  averages  provide  a  more  realistic 
picture  of  cost  growth. 
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C.   SYSTEM  COST  GROWTH 

1.  Total  Cost  Growth 

The  total  cost  growth  of  a  weapon  system  is  much  less  than  might 
be  expected.   It  has  a  dollar  weighted  average  value  of  0.108. 

A  1980  study  reports  that  the  average  total  cost  growth  of 
weapons  systems  is  25.9%  [Ref .  3] .   This  is  significantly  higher  than 
the  10.8%  this  analysis  shows.   However,  the  time  of  that  study  is 
relevant  to  the  difference  in  findings.   The  development  by  estimators 
of  a  computerized  large  data  base,  which  contains  the  production  costs 
of  a  large  number  of  components,  assemblies,  and  subassemblies,  allows 
for  a  more  accurate  estimate  of  the  production  costs  of  new  systems. 
This  is  beneficial  in  estimating  system  costs  which  should  bring  cost 
growth  down.   Another  cause  is  the  high  technologies  that  were  brand  new 
in  1980  and  extremely  difficult  to  estimate  have  been  developed  and 
produced  so  that  now  they  are  much  easier  to  estimate.   Lastly,  the 
results  of  the  1980  study  and  others  resulted  in  a  determined  effort  by 
the  Department  of  Defense  to  control  cost  growth.   This  has  also  had  a 

positive  impact  in  lowering  the  cost  growth  of  weapons  systems. 

2.  System  Model 

Eighty  five  percent  of  the  total  cost  growth  can  be  captured  in 
Dsmmi  and  the  Mcep  categories.   To  control  total  cost  growth,  it  is 
required  that  the  cost  growth  in  these  two  categories  be  focused  upon. 
The  possible  causes  of  the  cost  growth  in  these  categories  will  be 

discussed  below. 

3.  Mistake  and  Decision  Cost  Growth  Comparison 

Mistakes  make  up  30.6%  of  the  total  cost  growth  of  a  system 
while  decisions  make  up  6  9.4%  of  the  total  cost  growth.   Therefore 
decisions  outweigh  mistakes  by  a  margin  of  2.3:1. 

One  recent  study  reports  that  cost  growth  due  to  decisions 
outweighs  cost  growth  due  to  mistakes  by  a  margin  of  two  to  one  [Ref. 
7] .   This  analysis  reaches  a  similar  conclusion. 
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4.  Mistake  Cost  Growth 

A  majority  of  the  mistake  cost  growth  occurs  in  the  Mcep 
category.   The  Mcep  category  is  used  to  classify  cost  variances 
attributable  to  mistakes  in  estimating  the  cost  of  production.   While  it 
may  be  noted  that  this  implies  that  the  Department  of  Defense  needs  to 
concentrate  its  efforts  on  becoming  better  at  estimating  the  production 
costs  of  a  weapon  system,  the  mistake  cost  growth  for  a  system  is  only 
3.3%  and  is  not  nearly  as  significant  as  controlling  the  cost  growth  due 

to  decisions. 

5 .  Decision  Cost  Growth 

A  majority  of  the  decision  cost  growth  occurs  in  the  Dsmmi 
category.   The  Dsmmi  category  is  used  to  classify  cost  variances 
attributable  to  a  decision  to  change  the  procurement  schedule,  shifts  in 
the  multiyear  procurement  rate  or  in  different  management  initiatives. 
A  detailed  analysis  of  the  data  indicates  that  a  majority  of  the  Dsmmi 
cost  growth  is  due  to  schedule  slippage.   Some  of  this  schedule  slippage 
can  be  attributed  to  decisions  to  change  the  design  and  performance 
requirements  of  the  system.   These  changes  are  classified  into  the  Dcrv 
category  and  has  a  dollar  weighted  value  of  0.015.   This  may  or  may  not 
account  for  all  of  the  schedule  slippage,  however,  the  data  does  not 
provide  sufficient  evidence  to  support  or  deny  this  possible 
explanation.   Further  study  concentrating  on  the  causes  of  schedule 
slippage  needs  to  be  done. 

D.   SYSTEM  COST  COMPARISON 

The  driving  factor  between  the  difference  between  low  cost  system 
cost  growth  and  high  cost  system  cost  growth  is  the  difference  in  the 
mistake  cost  growth. 

The  low  cost  systems  have  a  higher  mistake  cost  growth  by  a  margin 
of  2.4:1.   Both  high  and  low  cost  systems  have  a  majority  of  their 
mistake  cost  growth  occurring  in  the  Mcep  category.   This  is  not 
surprising  since  this  result  has  already  been  determined  above. 
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However,  low  cost  systems  have  over  twice  as  much  cost  growth  in  the 
Mcep  category  than  do  high  cost  systems.   This  indicates  that  estimators 
are  not  nearly  as  adept  at  estimating  the  production  costs  of  low  cost 
systems  as  high  cost  systems. 

One  possible  explanation  for  this  is  that  estimators  do  not  take  as 
much  care  in  estimating  the  costs  of  low  cost  systems .   Low  cost  systems 
are  not  nearly  as  visible  on  a  Congressional  oversight  level  or  on  an 
internal  level,  and  therefore  the  care  taken  to  accurately  estimate  the 
production  cost  of  a  system  may  not  be  nearly  as  high  as  for  that  of  the 
more  expensive  systems. 

Another  explanation  concerns  the  nature  of  high  cost  to  low  cost 
systems  [Ref .  1] .   A  majority  of  high  cost  systems  are  large  platforms 
which  have  a  majority  of  their  production  cost  resulting  from  large 
components.   These  large  components,  such  as  hull  construction  or 
propulsion  plants,  have  been  used  before  or  a  similar  component  has  been 
used  before.   This  makes  the  estimation  of  the  production  costs  of  these 
large  components  fairly  accurate.   The  mistakes  in  estimating  the 
smaller  and  relatively  cheaper  items  would  be  masked  in  the  overall  cost 
growth  of  the  system  even  if  they  were  of  equal  dollar  value  to  the 
errors  in  the  low  cost  systems. 

Therefore,  the  difference  between  the  low  cost  and  high  cost  system 
mistake  cost  growth  is  probably  not  as  significant  as  it  appears  to  be. 

E.   MISSILE  SYSTEMS 

The  significant  difference  between  the  cost  growth  of  missile 
systems  as  compared  to  other  weapon  systems  is  that  the  mistake  cost 
growth  for  missile  systems  is  significantly  less  than  that  of  other 
systems,  specifically  in  estimating  the  cost  of  production.   The  most 
likely  explanation  is  similar  to  the  discussion  above  concerning  the 
cost  of  a  system.   Missiles  are  evolutionary  in  nature.   The  guidance 
system  of  newer  missiles  is  generally  a  modernization  or  modification  of 
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a  previous  guidance  system.   The  same  can  be  said  for  propulsion  system 
and  warhead  type.   This  would  tend  to  increase  the  accuracy  of  missile 
systems  over  other  types  of  systems. 

One  report  on  cost  growth  states  that  missile  systems  have  a  much 
higher  cost  growth  than  do  other  systems  [Ref .  1] .   This  article, 
prepared  in  1982,  included  much  older  missile  systems.   The  evolutionary 
process  discussed  above  had  not  yet  developed  and  therefore  a  higher 
cost  growth  could  have  resulted.   A  comparison  of  the  cost  growth  of 
older  missile  systems  to  newer  ones  would  be  required  to  validate  this 
assertion. 

F.   SERVICE  COMPARISON 

The  Army  and  Navy  have  effectively  equivalent  cost  growths.   The 
only  difference  is  that  part  of  the  Army's  mistake  cost  growth  results 
from  mistakes  in  the  estimation  process  that  resulted  in  schedule  slips. 
The  reasons  behind  this  are  not  clear  and  need  to  be  addressed  in  the 
future . 

A  majority  of  early  1980s  studies  reported  that  the  Army  had  a 
significantly  higher  cost  growth  than  the  Navy  [Refs.  1,  3] .   This  was 
attributed  to  the  modernization  of  the  Army.   This  study  concentrated  on 
more  recent  weapons  systems  and  did  not  find  this  to  be  the  case.   The 
most  obvious  explanation  for  this  effect  is  that  the  Army  has  completed 
its  modernization  and  is  now  proficient  at  controlling  the  costs  of  its 
weapon  systems . 
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VII .   RECOMMENDATIONS 

A.  NEW  CATEGORIES 

The  new  categorization  system  should  be  used  to  classify  cost 
variances  in  the  SAR  system.   This  would  allow  for  a  more  detailed 
accounting  of  the  cost  variances  and  allow  for  identification  of  the 
exact  areas  in  which  cost  growth  is  occurring. 

B.  TOTAL  COST  GROWTH 

The  total  cost  growth  as  been  significantly  lowered  over  the  past 
ten  years.   This  is  significant  and  promising.   The  Department  of 
Defense  should  continue  its  current  oversight  and  decision  process  to 
ensure  that  future  cost  growth  is  maintained  at  acceptable  levels. 

C.  DECISION  COST  GROWTH 

Decision  cost  growth  is  the  driving  force  behind  the  total  cost 
growth  of  weapon  systems.   This  is  driven  by  decisions  that  result  in 
schedule  slippage  in  the  procurement  process.   This  type  of  cost  growth 
must  be  minimized.   The  most  likely  solution  is  to  ensure  at  Milestone 
II  that  all  decisions  that  can  be  made  that  could  result  in  schedule 
slippage  are  made  and  finalized. 

D.  MISTAKE  COST  GROWTH 

The  mistake  cost  growth  is  well  within  any  reasonable  limits.   The 
driving  factor  for  this  type  of  growth  is  mistakes  in  estimating  the 
production  costs  of  a  weapon  system.   This  amounts  to  only  3.3%  cost 
growth  and  is  well  within  any  reasonable  margin  of  error  that  could  be 
placed  on  this  type  of  estimation. 

E.  ARMY  COST  GROWTH 

The  Army  mistake  cost  growth,  while  similar  in  magnitude  to  the 
Navy's,  is  partially  driven  by  schedule  slippage.   The  Army  must 
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evaluate  the  factors  that  are  causing  this  problem  and  correct  them. 
One  possible  solution  is  to  compare  the  Army' s  estimation  process  with 
the  Navy's  and  to  attempt  to  isolate  the  key  differences  that  result  in 
this  type  of  cost  growth. 

F.   FUTURE  STUDIES 

All  of  the  results  and  conclusions  of  this  study  are  tentative  since 
they  are  based  on  nine  weapon  systems.   This  is  the  initial  analysis  of 
a  three  year  study  by  the  DoD  and  the  results  and  conclusions  of  the 
study  using  all  of  the  SAR  reportable  weapon  systems  may  differ  from 
this  report. 

A  continuation  of  this  study,  once  data  for  more  weapons  systems  has 
been  obtained,  should  be  done  to  compare  Air  Force  systems  with  those  of 
the  Army  and  Navy.   Also,  a  comparison  of  aircraft  and  ship  cost  growth 
would  be  beneficial. 

A  study  of  RDT&E  cost  growth  is  necessary  to  evaluate  the  decision 
and  estimation  process  of  this  aspect  of  weapon  system  acquisition. 
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